Text mining

Text mining, sometimes alternately referred to as text data mining, roughly equivalent to text analytics, refers to the process of deriving high-quality information from text. High-quality information is typically derived through the devising of patterns and trends through means such as statistical pattern learning. Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output. 'High quality' in text mining usually refers to some combination of relevance, novelty, and interestingness. Typical text mining tasks include text categorization, text clustering, concept/entity extraction, production of granular taxonomies, sentiment analysis, document summarization, and entity relation modeling (i.e., learning relations between named entities).

Contents

History

Labor-intensive manual text mining approaches first surfaced in the mid-1980s, but technological advances have enabled the field to advance during the past decade. Text mining is an interdisciplinary field that draws on information retrieval, data mining, machine learning, statistics, and computational linguistics. As most information (common estimates say over 80%)[1] is currently stored as text, text mining is believed to have a high commercial potential value. Increasing interest is being paid to multilingual data mining: the ability to gain information across languages and cluster similar items from different linguistic sources according to their meaning.

Applications

Recently, text mining has received attention in many areas.

Security applications

Many text mining software packages are marketed for security applications, especially analysis of plain text sources such as Internet news. It also involves in the study of text encryption.

Biomedical applications

Main article: Biomedical text mining

A range of text mining applications in the biomedical literature has been described.[2]

The more important online text mining application in the biomedical literature is GoPubMed.[3] GoPubmed was actually the first semantic search engine on the Web. Other example is PubGene that combines biomedical text mining with network visualization as an Internet service.[4]

Software and applications

Text mining methods and software is also being researched and developed by major firms, including IBM and Microsoft, to further automate the mining and analysis processes, and by different firms working in the area of search and indexing in general as a way to improve their results. Within public sector much effort has been concentrated on creating software for tracking and monitoring terrorist activities.[5]

Online media applications

Text mining is being used by large media companies, such as the Tribune Company, to disambiguate information and to provide readers with greater search experiences, which in turn increases site "stickiness" and revenue. Additionally, on the back end, editors are benefiting by being able to share, associate and package news across properties, significantly increasing opportunities to monetize content.

Marketing applications

Text mining is starting to be used in marketing as well, more specifically in analytical customer relationship management. Coussement and Van den Poel (2008)[6] apply it to improve predictive analytics models for customer churn (customer attrition).[7]

Sentiment analysis

Sentiment analysis may involve analysis of movie reviews for estimating how favorable a review is for a movie.[8] Such an analysis may need a labeled data set or labeling of the affectivity of words. Resources for affectivity of words and concepts have been made for WordNet[9] and ConceptNet,[10] respectively.

Text has been used to detect emotions in the related area of affective computing ,[11] e.g., sentic computing.[12] Text based approaches to affective computing have been used on multiple corpora such as students evaluations, children stories and news stories.

Academic applications

The issue of text mining is of importance to publishers who hold large databases of information needing indexing for retrieval. This is especially true in scientific disciplines, in which highly specific information is often contained within written text. Therefore, initiatives have been taken such as Nature's proposal for an Open Text Mining Interface (OTMI) and the National Institutes of Health's common Journal Publishing Document Type Definition (DTD) that would provide semantic cues to machines to answer specific queries contained within text without removing publisher barriers to public access.

Academic institutions have also become involved in the text mining initiative:

Notable software and applications

Text mining computer programs are available from many commercial and open source companies and sources.

Commercial

Free libre open-source

  1. Carrot2 – text and search results clustering framework.
  2. GATE – natural language processing and language engineering tool.
  3. OpenNLP - natural language processing
  4. Natural Language Toolkit (NLTK) – a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for the Python programming language.
  5. RapidMiner with its Text Processing Extension – data and text mining software. Rated as the fifth most used text mining software (6%) by Rexer's Annual Data Miner Survey in 2010.[16]
  6. Unstructured Information Management Architecture (UIMA) – a component framework to analyze unstructured content such as text, audio and video, originally developed by IBM.
  7. Knime - Open Source Data Mining Tool with an experimental Textprocessing-Extension [18]

Implications

Until recently, websites most often used text-based searches, which only found documents containing specific user-defined words or phrases. Now, through use of a semantic web, text mining can find content based on meaning and context (rather than just by a specific word).

Additionally, text mining software can be used to build large dossiers of information about specific people and events. For example, large datasets based on data extracted from news reports can be built to facilitate social networks analysis or counter-intelligence. In effect, the text mining software may act in a capacity similar to an intelligence analyst or research librarian, albeit with a more limited scope of analysis.

Text mining is also used in some email spam filters as a way of determining the characteristics of messages that are likely to be advertisements or other unwanted material.

See also

Notes

  1. ^ Unstructured Data and the 80 Percent Rule
  2. ^ K. Bretonnel Cohen & Lawrence Hunter (January 2008). "Getting Started in Text Mining". PLoS Computational Biology 4 (1): e20. doi:10.1371/journal.pcbi.0040020. PMC 2217579. PMID 18225946. http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.0040020. 
  3. ^ GoPubMed: exploring PubMed with the Gene Ontology, A. Doms and M. Schroeder, 2005, http://nar.oxfordjournals.org/content/33/suppl_2/W783.long
  4. ^ Tor-Kristian Jenssen, Astrid Lægreid, Jan Komorowski1 & Eivind Hovig (2001). "A literature network of human genes for high-throughput analysis of gene expression". Nature Genetics 28 (1): 21–28. doi:10.1038/ng0501-21. PMID 11326270. http://www.nature.com/ng/journal/v28/n1/abs/ng0501_21.html. 
  5. ^ Texor
  6. ^ Academic Papers about Analytical Customer Relationship Management
  7. ^ Kristof Coussement, and Dirk Van den Poel (forthcoming 2008). "Integrating the Voice of Customers through Call Center Emails into a Decision Support System for Churn Prediction". Information and Management. http://www.textmining.ugent.be. 
  8. ^ Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan (2002). "Thumbs up? Sentiment Classification using Machine Learning Techniques". Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 79–86. http://www.cs.cornell.edu/home/llee/papers/sentiment.pdf. 
  9. ^ Alessandro Valitutti, Carlo Strapparava, Oliviero Stock (2005). "Developing Affective Lexical Resources". Psychology Journal 2 (1): 61–83. http://www.psychnology.org/File/PSYCHNOLOGY_JOURNAL_2_1_VALITUTTI.pdf. 
  10. ^ Erik Cambria; Robert Speer, Catherine Havasi and Amir Hussain (2010). "SenticNet: a Publicly Available Semantic Resource for Opinion Mining". Proceedings of AAAI CSK. pp. 14-18. http://www.aaai.org/ocs/index.php/FSS/FSS10/paper/download/2216/2617.pdf. 
  11. ^ Rafael A. Calvo, Sidney K. D'Mello (2010). "Affect Detection: An Interdisciplinary Review of Models,Methods, and their Applications". IEEE Transactions on Affective Computing 1 (1): 18–37. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=5520655. 
  12. ^ Cambria, Erik (2012). Roelandse, Martijn. ed. Sentic Computing: Techniques, Tools and Applications. Berlin: Springer-Verlag. 
  13. ^ The University of Manchester
  14. ^ Tsujii Laboratory
  15. ^ The University of Tokyo
  16. ^ a b c d Rexer Analytics 4th Annual Data Miner Survey - 2010
  17. ^ IBM - SPSS - Software Products
  18. ^ http://www.tech.knime.org/knime-text-processing-0

References

External links